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Multi-level selection methods and apparatus using context identification 



(57) An image capture user interface receives an 
image of an area of a user interface selected by a user 
and translates the image into operations performable by 
a computer. The user interface is comprised of graphic 
entities and embedded code. The user places an image 
capture device, such as a camera pen, on or near a 
graphic entity of the user interface, and presses a button 
on the image capture device indicating selection of the 
graphic entity. In response to the button, an image is 
captured that corresponds to the graphic entity selected 
by the user. The image includes embedded code, which 



is analyzed to develop an image capture code corre- 
sponding to the captured image area. The image cap- 
ture code is then mapped to a selection code corre- 
sponding to the graphic entity selected by the user. The 
user may then make other selections. The selection 
codes are processed for a particular syntax, and a com- 
puter operation is performed when a selection code, or 
combination of selection codes, is received which indi- 
cate that an operation is to be performed. In other em- 
bodiments, mapping of image capture codes to selec- 
tion codes and syntax processing may be performed in 
accordance with a particular context. 
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Description 

[0001] Conventional user interfaces allow users to se- 
lectively initiate computer system operations. A compu- 
ter coordinates an electronic display and a user input s 
device, such as a keyboard. Command line interfaces, 
for example, require users to type in commands direct- 
ing the computer to perform a particular operation. 
Graphical user interfaces relieve users of the burden of 
typing commands by allowing a graphical selection of 10 
computer system entities and computer system opera- 
tions. For example, when the user uses a mouse to se- 
lect a file icon on a screen, the computer accesses the 
file associated with the icon. Graphical user interfaces 
are particularly advantageous because they are intuitive is 
and therefore easy to use. 

[0002] Heretofore, conventional graphical user inter- 
faces were configured to permit electronic (X, Y) ad- 
dressing of graphic elements displayed on a computer 
I/O device, such as a screen of a computer monitor. The 20 
address space accessible at any given time is con- 
strained by the display space. Larger address spaces 
can be implemented within the constraints of the display, 
but this requires the user to navigate sequentially 
through the larger space within the dimensions of the 2s 
display. 

[0003] A conventional graphical user interface is also 
constrained by the requirement that a computer system 
must coordinate electrical signals from user input devic- 
es with the user interface graphics on the display. This 30 
limits flexibility because the computer must produce the 
user interface on the display, and coordinate input re- 
ceived from the input devices. 

[0004] What is needed, then, is a computer interface 
that is not constrained by the typical dimensions of an 35 
electronic display, and does not require electronic coor- 
dination between the display device and the user input 
device. 

[0005] Apparatus and methods consistent with the 
principles of the invention provide a user interface that 40 
utilizes multi-level image capture and context identifica- 
tion. An apparatus consistent with the invention, for use 
with a user interface having sensory indicia and embed- 
ded data code, and an image capture device for select- 
ing a portion of the user interface and sending image 45 
information corresponding to the portion, comprises an 
image processor for decoding first image information in- 
to a first image capture code, a resolver for mapping the 
first image capture code into a first selection code, and 
a syntax processor for analyzing the first selection code so 
with a second selection code. 

[0006] A method consistent with the invention, for use 
with a user interface having sensory indicia and embed- 
ded data code, and an image capture device for select- 
ing a portion of the user interface and sending image ss 
information corresponding to the portion, comprises de- 
coding first image information into a first image capture 
code, mapping the first image capture code into a first 



selection code and analyzing the first selection code 
with a second selection code. 

[0007] Human sensory information, such as textual, 
graphical or mixed textual and graphical representa- 
tions of the computer system entities and operations 
may be selected using this user interface. The sensory 
information is spatially registered in superimposed or 
juxtaposed relationship with respective spatial address- 
es in the embedded data address space. These spatial 
addresses, in turn, are registered in a lookup table or 
similar mechanism with computer recognizable descrip- 
tions of the respective computer system operations or 
entities. 

[0008] Still further objects of this invention will be- 
come apparent when the following detailed description 
is read in conjunction with the attached drawings, in 
which: 

Fig. 1 illustrates a self-clocking glyph code pattern 

and a portion of its binary interpretation; 

Fig. 2 is a block diagram that shows an example of 

a user interface implemented using glyph address 

carpets; 

Fig. 3 is a block diagram of an image capture sys- 
tem that may be used to capture a user-selected 
portion of the glyph address carpet, and decode the 
glyphs in the captured portion; 
Fig. 4 shows an embodiment of a user interface im- 
plementing hot zones; 

Fig. 5 is a block diagram illustrating the selection 
process using camera pen 1710; 
Fig. 6 is a block diagram illustrating a second type 
of selection process that may be used to implement 
a graphical user interface consistent with the prin- 
ciples of the invention 

Fig. 7 illustrates an embodiment of the address 
codes encoded in the glyph address carpet 1732; 
Fig. 8 and Fig. 9 form a flow chart showing the dis- 
ambiguation and address decoding processes per- 
formed on the captured image area by computer 
1712; 

Fig. 1 0 illustrates a binary data matrix 231 0 formed 
from a glyph lattice captured by camera pen 1710; 
Fig. 11 is a flowchart showing the process for per- 
forming correlation steps 2116 and 2118; 
Fig. 12 illustrates the analysis performed by com- 
puter 1712 to determine the address encoded by 
the glyphs of the captured image; 
Fig. 13 illustrates how the address carpet 1612 is 
divided into areas having respective ranges of ad- 
dresses. 

Fig. 14 illustrates a user interface 3210 having sev- 
eral icons that a user can select individually using 
camera pen 1710; 

Fig. 15 illustrates an icon overlaying embedded 
codes; 

Fig. 1 6 is a block diagram illustrating the processes 
of computer 1712 that implement image capture, 
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embedded data decoding, code resolving, and syn- 
tax decoding; 

Fig. 17 is a flowchart showing the processing per- 
formed by computer 1712 when receiving a selec- 
tion from the user; 

Fig. 18 is a flow chart showing the processing per- 
formed by computer 1712 in response to decoded 
embedded data information; 
Fig. 19 is a block diagram of the elements used by 
computer 1712 when determining whether the 
pending codes collectively represent an operation 
to be performed; 

Figs. 20 and 21 illustrate user interfaces 3810 and 
3812, respectively, each having respectively differ- 
ent contexts, 

Figs. 22, 23, 24, and 25 show logical representa- 
tions of embodiments of code resolver 3412 and 
syntax processor 3414; 

Fig. 26 illustrates a state diagram showing process- 
ing of incoming codes using context-based syntax 
processing; 

Fig. 27 illustrates a user interface embodiment sim- 
ilar to the interface shown in Fig. 14, but is imple- 
mented using explicitly marked hot zones around 
the graphic entities; 

Fig. 28 is a block diagram illustrating a variety of 
different user interfaces which may be used to se- 
lect various computer system entities and functions; 
Fig. 29 illustrates the principles of the invention in 
a document processing environment including a 
text document; and 

Fig. 30 illustrates a user interface example showing 
an e-mail system using local embedded data code 
labels. 

[0009] Turning now to the drawings, and at this point 
especially to Fig. 1 , there is a more or less conventional 
self-clocking glyph code pattern 21 which is composed 
of elongated slash-like marks or "glyphs ■ 22 and 23 that 
are written on a generally regular rectangular lattice of 
centers on a suitable recording medium 24. Suitably, the 
glyphs 22 and 23 are printed by a printer (not shown) 
operating at 300 d.p.i. - 600 d.p.i. to write 4 pixel x 4 
pixel - 7 pixel x 7 pixel representations of the glyphs 22 
and 23 on regularly spaced centers that are distributed 
widthwise and lengthwise of the recording medium 24 
to produce the rectangular code pattern 21 . The glyphs 
of these fine grain glyph code patterns are not easily 
resolved by the unaided human eye when the code pat- 
terns are viewed under standard lighting conditions and 
at normal reading distances, so the code pattern 21 typ- 
ically has a generally uniform gray scale appearance. 
Nevertheless, the glyph code is still capable of effective- 
ly communicating machine readable digital information. 
To carry out this function, the glyphs 22 and 23 usually 
are titled to the left and right, at about +45° and -45° with 
respect to the longitudinal dimension of the recording 
medium 24 to encode binary "Vs" and "O's", respective- 



ly, as shown at 25. 

[0010] Fig. 2 is a block diagram that shows an exam- 
ple graphical user interface implemented using a glyph 
address carpet. The interface comprises two compo- 
5 nents: (1) sensory indicia and (2) a background which 
comprises a glyph address carpet 1612. The address 
carpet may be visible or invisible. The sensory indicia in 
Fig. 2 are preferably visual indicia, and in particular 
graphic elements, such as icon 1 61 0. The glyph address 
io carpet 1612 is used as the background wallpaper for the 
graphic elements. The glyph pattern that makes up the 
background is called an "address carpet" because the 
glyphs can be decoded to provide unique address infor- 
mation for each location. 
is [0011] The icons may, as shown in Fig. 2, be partially 
formed by glyphs that are part of the glyph address car- 
pet. In the case of glyphtones (see, e.g., U.S. Patent No. 
5,315, 098toTow)andserpentones (see, e.g., U.S. Pat- 
ent No. 5,706,099 to Curry), the indicia may be substan- 
20 tially composed of the glyphs themselves. Because the 
glyphs encode address information, a portion of the 
glyphs can be read optically and decoded to determine 
a unique address for a local position. 
- [0012] The graphical user interface implemented us- 
25 ing glyph address carpets can be implemented on any 
media capable of displaying the components of the in- 
terface. Thus, the interface could be implemented on 
hardcopy, such as paper, labels, physical objects, and 
photographic media; on dynamic displays such as a 
so cathode ray tube (CRT) display or liquid crystal display 
(LCD); or on projected displays from fixed or dynamic 
media such as a slide projector, or television. 
[0013] Although the embodiment described herein 
utilizes glyphs, the glyphs can alternatively be replace^ 
35 by any system, visible or invisible, that provides a data 
address code. Whereas the embodiment described 
herein utilizes visual indicia, the visual indicia can alter- 
natively be replaced by any sensory indicia that could 
serve to guide a user or machine to select a location 
40 while the machine readable data address codes provide 
an address logical reference. The embodiment de- 
scribed herein utilizes a paper substrate, but the paper 
substrate can alternatively be replaced by any media in 
which the data address codes can be embedded and 
45 read by a machine. 

[0014] The sensory indicia may be embedded in the 
same media as the data address codes or may be em- 
bedded in other media. The sensory indicia can alterna- 
tively be projected onto glyphs, or glyphs can be pro- 
50 jected onto sensory indicia. 

[0015] Fig. 3 is a block diagram of an image capture 
system that may be used to capture a user-selected por- 
tion of a graphical user interface, such as glyph address 
carpet 1 2, and decode the glyphs in the captured por- 
55 tion. In one embodiment, computer system 1712 is a 
general purpose computer system, such as a conven- 
tional personal computer or laptop computer, that in- 
cludes main memory 1716, read only memory (ROM) 
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1718, storage device 1720, processor 1722, and com- 
munication interface 1724, all interconnected by bus 
1726. Bus 1726 also connects to display 1730, cursor 
control 1714, and frame capture 1728. 
[0016] Image capture device 1710, which in this case 
is a camera pen, is connected to frame capture 1728 
and mouse 1730. Camera pen 1710 transmits image in- 
formation to frame capture 1728. In one embodiment, 
button 1 71 4 of camera pen 1 71 0 is wired to mouse 1 730 
so that when a user presses button 1714a signal travels 
through the circuitry of mouse 1728 to cursor control 
1714. The signal causes processor 1722 to run a pro- 
gram that directs frame capture 1728 to capture the im- 
age from camera pen 1710. In another embodiment, 
both the image line and signal line from camera pen 
1710 are input directly into frame capture card 1728. 
The lines between camera pen 1710 and computer 
1712 can be wired in any way that provides capture of 
the image from camera pen 1710. 
[0017] The user makes a selection by placing camera 
pen 1 71 0 on or near visual indicia on glyph address car- 
pet 1732, and pressing button 1714. Pressing button 
1714 causes camera pen 1 7 1 0 to capture the portion of 
the address carpet under the tip of camera pen 1710, 
and transmit the image to computer 1712, via frame cap- 
ture 1728, for analysis. The button or multiple buttons 
can be used for additional signaling, as in a double click, 
hold down. 

[0018] Fig. 4 shows an embodiment of a user inter- 
face implementing hot zones. A hot zone is an area on 
or near the visual indica that defines selection locations 
that are equivalent to selecting the same visual indicia. 
In a preferred embodiment, the hot zone may encom- 
pass the visual indicia. For example, in Fig. 4, the icon 
David's DOC2 has a broken line area indicating a hot 
zone. If the user makes a selection within this hot zone, 
the icon "David's DOC2" will be selected. This allows 
the user to make a selection that is on or near the visual 
indicia. The selection is treated by the system as if the 
visual indicia was selected. The hot zone may be explic- 
itly denoted by the visual indicia, including a visually dis- 
tinct coloring in the address carpet, or may be implicit 
by appropriate proximity. 

[0019] Fig. 4 also illustrates a preferred embodiment 
for the glyph address carpet coding scheme. Each glyph ■ 
is either a forward slash or a backward slash. The ori- 
entation and spacing of the rows and columns of glyphs 
is indicated by O gx and O gy , respectively As denoted 
by the As and Bs in the drawing, an A code runs on every 
other line, and is interlaced with a B code. Along diago- i 
nals running down and to the right, every A value is pref- 
erably the same. Similarly, along diagonals running 
down and to the left, every B value is preferably the 
same. 

[0020] Fig. 5 is a block diagram illustrating the selec- s 
tion process using camera pen 1710. Each icon has an 
effective selection area indicated by a broken line hot 
zone 1910. The tip of camera pen 1710 covers an area 



indicated by 1914. The orientation of the tip of camera 
pen 1 7 1 0 is denoted by the Y c axis and Xc axis. To make 
a selection, the user places the tip of camera pen 1710 
over the area to be selected. When the user presses 
5 button 1714, the image within area 1914 is captured. 
Computer 1712 analyzes the captured image to deter- 
mine the location of the center 1912 of selection area 
1914. After determining the location of center 1912, the 
location of center 1 91 2 is used to look up a function cor- 
io responding to center 1912. 

[0021] Fig. 6 is a block diagram illustrating a second 
type of selection process that may be used to implement 
a graphical user interface consistent with the principles 
of the invention. In this embodiment, camera pen 1710 
1* has a pointing device attached to the tip, thus making 
the image.capture area of the pen offset from the loca- 
tion pointed at by the user. For example, in Fig. 6, a user 
selects the icon "David's DOC" by pointing to the icon, 
but the camera pen 1710 image capture area 201 4, with 
7 *o center 2012, is offset from the icon. In this case, com- 
puter 1712 must determine the actual selection based 
on 1 ) the image area 201 4, and 2) the orientation of the 
selected area, and 3) the offset distance and direction 
from the image capture area from center 2012. The off- 
y s set calculation utilizes the glyph lattice parameters from 
the captured image decoding process described below. 
[0022] Returning to Fig. 3, in one embodiment, main 
memory 1716 is a random access memory (RAM) or a 
dynamic storage device that stores instructions execut- 
o ed by processor 1722. Main memory 1716 may also 
store information used in executing instructions. ROM 
1718 is used for storing static information and instruc- 
tions used by processor 1722. Storage device 1720, 
such as a magnetic or optical disk, also stores instruc- 
5 tions and data used in the operation of computer system 
1712. 

[0023] Display 1730 may be a CRT or other type of 
display device. Cursor control 1714 controls cursor 
movement on display 1730. Cursor control 1714 may 
> be, for example, a mouse, a trackball or cursor direction 
keys. 

[0024] The system shown in Fig. 3 can be used to im- 
plement the glyph address carpet capture and transla- 
tion system described herein. The apparatus and meth- 
: ods described herein may be implemented by computer 
system 1712 using hardware, software, or a combina- 
tion of hardware and software. For example, the appa- 
ratus and methods described herein may be implement- 
ed as a program in any one or more of main memory 
1716, ROM 1718, or storage device 1720. In one em- 
bodiment, processor 1722 executes programs which 
analyze captured portions of a glyph address carpet to 
determine address information encoded in the glyphs. 
[0025] Such programs may be read into main memory 
171 6 from another computer- readable medium, such as 
storage device 1 720. Execution of sequences of instruc- 
tions contained in main memory 171 6 causes processor 
1722 to perform the process steps consistent with the 
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present invention described herein. Execution of se- 
quences of instructions contained in main memory 1 716 
also causes processor to implement apparatus ele- 
ments that perform the process steps. Hard-wired cir- 
cuitry may be used in place of or in combination with 
software instructions to implement the invention. Thus, 
embodiments of the invention are not limited to any spe- 
cific combination of hardware circuitry and software. 
[0026] The term "computer-readable medium" as 
used herein refers to any medium that participates in 
providing instructions to processor 1722 for execution. 
Such a medium may take many forms, including but not 
limited to, non-volatile memory media, volatile memory 
media, and transmission media. Non-volatile memory 
media includes, for example, optical or magnetic disks, 
such as storage device 1720. Volatile memory media 
includes RAM, such as main memory 1716. Transmis- 
sion media includes coaxial cables, copper wire and fib- 
er optics, including the wires that comprise bus 1726. 
Transmission media can also take the form of acoustic 
or light waves, such as those generated during radio- 
wave and infrared data communications. 
[0027] Common forms of computer-readable media 
include, for example, a floppy disk, a flexible disk, hard 
disk, magnetic tape, or any other magnetic storage me- 
dium, a CD-ROM, any other optical medium, punch- 
cards, papertape, any other physical medium with pat- 
terns of holes, a RAM, a PROM, an EPROM, a FLASH- 
EPROM, any other memory chip or cartridge, a carrier 
wave as described hereinafter, or any other medium 
from which a computer can read and use. 
[0028] various forms of computer readable media 
may be involved in carrying one or more sequences of 
instructions to processor 1722 for execution. For exam- 
ple, the instructions may initially be carried on a mag- 
netic disk or a remote computer. The remote computer 
can load the instructions into its dynamic memory and 
send the instructions over a telephone line using a mo- 
dem. A modem local to computer system 1712 can re- 
ceive the data on the telephone line and use an infrared 
transmitter to convert the data to an infrared signal. An 
infrared detector coupled to appropriate circuitry can re- 
ceive the data carried in the infrared signal and place 
the data on bus 1 726. Bus 1 726 carries the data to main 
memory 1716, from which processor 1722 retrieves and 
executes the instructions: The instructions received by 
main memory 1716 may optionally be stored on storage 
device 1720 either before or after execution by proces- 
sor 1722. 

[0029] Computer system 1712 also includes a com- 
munication interface 1724 coupled to bus 1726. Com- 
munication interface 1724 provides two way communi- 
cations to other systems. For example, communication 
interface 1 724 may be an integrated services digital net- 
work (ISDN) card or a modem to provide a data com- 
munication connection to a corresponding type of tele- 
phone line. Communication may also be, for example, 
a local area network (LAN) card to provide communica- 



tion to a LAN. Communication interface 1724 may also 
be a wireless card for implementing wireless communi- 
cation between computer system 1712 and wireless 
systems. In any such implementation, communication 

5 interface 1724 sends and receives electrical, electro- 
magnetic or optical signals that carry data streams rep- 
resenting various types of information. 
[0030] The link between communication interface 
1724 and external devices and systems typically pro- 

io vides data communication through one or more net- 
works or other devices. For example, the link may pro- 
vide a connection to a local network (not shown) to a 
host computer or to data equipment operated by an In- 
ternet Service Provider (ISP). An ISP provides data 

'5 communication services through the world wide packet 
data communications network now commonly referred 
to as the "Internet*. Local networks and the Internet both 
use electrical, electromagnetic or optical signals that 
carry digital data streams. The signals through the var- 
20 ious networks and the signals between the networks 
and communication interface 1724, which carry the dig- 
ital data to and from computer system 1712, are exem- 
plary forms of carrier waves transporting the informa- 
tion.- 

25 [0031] Computer system 1712 can send messages 
and receive data, including program code, through the 
network(s) via the link between communication inter- 
face 1 724 and the external systems and devices. In the 
Internet, for example, a server might transmit a request- 

30 ed code for an application program through the Internet, 
an ISP, a local network, and communication interface 
1724. 

[0032] Program code received over the network may 
be executed by processor 1 722 as it is received, and/or 
35 stored in memory, such as in storage device 1 720, for 
later execution. In this manner, computer system 1712 
may obtain application code in the form of a carrier 
wave. 

[0033] Fig. 7 illustrates an embodiment of the address 

40 codes encoded in the glyph address carpet 1 732. More 
particularly, Fig. 7 illustrates one section 2110 of the 
glyph address carpet. The addresses are encoded by 
alternating rows of "A" address code sequences and "B" 
address code sequences. The position along each se- 

46 quence in each row should be unambiguously determi- 
nable from a predetermined length subsequence. For 
example, an N-bit shift register maximal length code can 
be uniquely determined in position from an N-bit subse- 
quence. Each address code sequence is a fifteen bit se- 

so quence, with the A sequence indexed running left to 
right, and the B sequence indexed running in a reverse 
direction, right to left. Each row of A code sequences is 
offset by two glyph positions relative to the previous and 
next row of A addresses. Similarly, each row of B code 

55 sequences is offset in the opposite direction by two po- 
sitions. Thus, the encoding scheme has two key char- 
acteristics: parallel rows including two sets of one-di- 
mensional unique address codes and relative offsets 
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among members of the two sets so that the offset be- 
tween each pair from the two sets is unique. This estab- 
lishes two-dimensional unique address locations. 
[0034] Computer 1712 decodes address information 
encoded in the glyphs by analyzing the captured image 
area in two steps. Ideally, the user places camera pen 
1710 over a portion of glyph address carpet 1732 and 
capture an image angularly aligned as shown in the pat- 
tern of bits shown in Fig. 7. In reality, however, the user 
variably orients the camera pen 1710 over an area of 
interest, so the pattern could be oriented anywhere from 
0° to 360°. Therefore, the computer 1712 must first de- 
termine the orientation of the image as part of decoding 
and interpreting the address information. 
[0035] The orientation of the image is determined by 
analyzing the captured image. This process is called 
disambiguation. (See, e.g., U.S. Patent No. 5,521,372 
to Hecht, et al.). After determining the proper orientation 
of the image, computer 1712 decodes the address of 
the selected location in the address carpet The disam- 
biguation and address decoding processes performed 
by computer 1712 will now be described in greater de- 
tail. 

[0036] Fig. 8 and Fig. 9 form a flow chart showing the 
disambiguation and address decoding processes per- 
formed by computer 1712 on the captured image area. 
Computer 1712 begins the disambiguation process by 
image processing the captured portion of the address 
carpet to find a glyph seed. A glyph seed is a first-iden- 
tified glyph having readable glyphs around it. Once a 
glyph seed has been found, the glyph lattice can be de- 
termined by processing neighbors of the glyph seed. 
(See Appendices A and B). The glyphs are then decod- 
ed as 1 's or 0's, which are filled into a binary data matrix 
having rows and columns corresponding to the glyph lat- 
tice rows. The orientation may still be ambiguous with 
respect to 90° and 180° rotations. 
[0037] Fig. 10 illustrates a binary data matrix (BDM) 
2310 formed from a glyph lattice captured by camera 
pen 1 71 0. The BDM has locations corresponding to the 
glyph lattice. Thus the size of the BDM corresponds 
closely to the size of the glyph lattice. 
[0038] Each location of the glyph lattice is analyzed 
to determine which value should be placed in the corre- 
sponding location of the BDM. Initially, the BDM is filled 
with a value, for example <|>, which indicates that no at- 
tempt has been made to read the glyph. Once the glyph 
corresponding to a particular location has been ana- 
lyzed, the $ is replaced by a value indicating the result 
of the glyph analysis. 

[0039] In Fig. 10, a B indicates a border location; an 
X indicates that no interpretable glyph was found at the 
corresponding location of the glyph lattice; an E indi- 
cates a glyph at the edge of the captured image portion; 
a 0 indicates a back slash glyph; and a 1 indicates a * 
forward slash glyph. The area of the matrix correspond- 
ing the captured image is filled with 0's and 1 's, the edge 
is bounded by Es, and the Xs correspond to locations 



that have no readable glyphs. In practice, however, al- 
though the BDM will generally have a similar pattern, 
the values will often not be as evenly distributed. For 
example, a glyph location within the captured image ar- 
s ea might result in an X if the glyph has been obliterated. 
Several values have been drawn with circles and 
squares around them to illustrate the two separate code 
sequences that are staggered in opposite directions. 
[0040] When a user makes a selection, the user might 
io orient the camera pen in virtually any direction on the 
user interface. The captured image could be oriented at 
any angle. Thus, even though step 2210 allows compu- 
ter 1 71 2 to derive a BDM of 0's and 1 's from the captured 
image, it is uncertain whether the BDM is oriented at 0° 
is (i.e., correctly oriented), 90°, 180°, or 270° relative to 
the original code pattern in the glyph address carpet 
from which the image was captured. Until the orientation 
of the captured image is determined, it is not possible 
to derive the correct address code from the BDM. The 
20 orientation could be provided by auxiliary information 
such as physical system constraints. However, the ori- 
entation can be uniquely determined directly from the 
address codes. 

[0041] After converting the glyphs to 0's and 1 's, a ref- 
?5 erence glyph location is selected (step 221 1 ). This loca- 
tion may be chosen in a variety of ways, but is typically 
a location which represents the selection. For example, 
the reference glyph location could be at the center of the 
BDM. 

[0042] After the image has been converted to a BDM, 
it is processed by computeM 71 2 (step 221 2). The orig- 
inal BDM developed from the captured image is referred 
to as BDM1. Computer 1712 makes a copy of BDM1 
and rotates the copy clockwise 90° to form a second 

55 binary data matrix, BDM2 (step 221 4). By rotating BDM1 
by 90°, the rows of BDM1 become the columns of 
BDM2, and the columns of BDM1 become the rows of 
BDM2. Additionally, all bit values in BDM2 are flipped 
from 0 to 1 , and 1 to 0, because a 45° slash glyph rotated 

40 90° appears as the opposite state of the non-rotated 
glyph. 

[0043] Computer 1712 then performs a correlation 
separately on the odd and even rows of BDM1 (step 
2216) to determine whether code in the rows are stag- 

45 gered forward or backward. The correlation is also per- 
formed for the odd and even rows of BDM2 (step 2218). 
The correlation is performed over all the rows of each 
BDM, and results in correlation value C1 for BDM1 and 
correlation value C2 for BDM2. 

so [0044] Fig. 11 is a flowchart showing an embodiment 
of correlation steps 2216 and 2218 of Fig. 9. The proc- 
ess determines a correlation value for every other line 
of a BDM along diagonals in each direction, and sums 
the row correlation values to form a final correlation val- 

>5 ue for the odd or even rows. The process is performed 
on the odd rows of BDM1 to form correlation value 
C1qdd for BDM1, the even rows of BDM1 to form cor- 
relation value C1 even for BDM1 , the odd rows of BDM2 
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to form correlation value C2 ODD for BDM2, the even 
rows of BDM2 to form correlation value C2 EVEN for 
BDM2. The BDM that is oriented at 0° or 180° will have 
a larger C ODD + C EVEN than the other BDM. 
[0045] Computer 1712 first inputs the BDM (step 
2410), and then copies every other row to a temporary 
matrix (step 2412). Identical processes are then per- 
formed for diagonals to the right and to the left. Steps 
241 4, 241 6, 241 8, 2420, 2422 and 2424 process the di- 
agonals to the right. For example, in Fig. 12 the steps 
correlate along the diagonals moving from the upper left 
to lower right. First, row count N and correlation value 
C_RIGHT are each initialized to zero (step 2414). Row 
N is shifted two places to the right, and correlated with 
the next row (step 2416). CJM is then set to this value 
(step 2418). CJ=tlGHT is then set to C_RIGHT + C_N 
(step 2420), and N is incremented (step 2422). If row 
count N >= Nmax, where N is the number of odd or even 
rows in the BDM, then the process proceeds to step 
2426. If N is not greater than or equal to Nmax, the proc- 
ess continues at step 241 6. Thus, after the process has 
correlated each adjacent row, the correlation value 
C_RIGHT indicates the strength of the correlation along 
the diagonals to the right. 

[0046] The steps on the right of Fig. 11 are similar to 
steps 241 4, 241 6, 241 8, 2420, 2422 and 2424, but proc- 
ess diagonals running from the upper right to lower left 
to develop correlation value C_LEFT. After correlating 
the right and left diagonals to determine C_RIGHT and 
C_LEFT, a final correlation value C is determined by 
subtracting CJ.E FT from C_RIGHT. For example, if odd 
rows for BDM1 are processed, the C value becomes 
C1ooD tor BDM1. 

[0047] The process steps of Fig. 1 1 are performed for 
the odd and even rows of BDM1 and the odd and even 
rows of BDM2. From this information, the correlation val- 
ue C1 for BDM1 is set to Cl^gs, + C1 ODD (as deter- 
mined by Fig. 11 for the rows of BDM1), and the corre- 
lation value C2 for BDM2 is set to C2 EVEN + C2 ODD (as 
determined by Fig. 11 for the rows of BDM1). 
[0048] Fig. 12 illustrates why the correlations deter- 
mine which way the codes in every other row are shift- 
ing. For example, as indicated by the circled A1s along 
the diagonals running to the right, the codes along the 
diagonal starting at A1 in the first position of the second 
row should have the same value in every other row 
along the diagonal, except for erasures and errors. Sim- 
ilarly, as indicated by the boxed B1s, the codes along 
the diagonal starting at B1 in the upper right corner 
should have the same value in every other row along 
the diagonal, except for erasures or errors. This is true 
for each value along the diagonal in the odd rows run- 
ning respectively from B2, B3, ... in the top row. Thus, 
the strong correlations along the diagonals running 
down and to the left on the odd rows, and the strong 
correlations along the diagonals running down and to 
the right on the even rows indicate that the codes in the 
even rows are shifting to the right, and the codes in the 



odd rows are shifting to the left. 
[0049] For each BDM, therefore, four correlation val- 
ues are developed: 1 ) odd rows, right to left, 2) odd rows, 
left to right, 3) even rows, right to left and 4) even rows, 
s left to right. From these correlation values, the strongest 
correlation value for the even rows, and strongest cor- 
relation value for the odd rows is chosen, and these be- 
come C even and c odd for tnat BDM (steps 2216 and 
2218). and C ODD are then added to form a final 

10 c correlation value for that BDM. As discussed above 
with respect to step 2220, the BDM with the strongest 
correlation value is the BDM that is oriented at either 0° 
or 180° because of the relative orientation of the codes 
in the odd and even rows. Thus, two aspects of the cho- 
is sen BDM are now established: which direction every 
other line of codes is staggered, and that the BDM is 
oriented horizontally, at either 0° or 180°. Another cor- 
relation process, step 2230 is performed to determine 
which direction the code in each line runs (as opposed 
20 to which way the code is staggered). 

[0050] The codes in the odd lines are staggered in one 
direction, and the codes in the even lines are staggered 
in the other. This staggering property of the code, in con- 
junction with knowing the respective codes that run in 
25 the odd lines and even lines, allows determination of the 
proper 0° orientation of the BDM. 
[0051] Returning to Fig. 8, if C1 is greater than C2 
(step 2220), then BDM1 is selected for further process- 
ing. C1 being greater than C2 indicates that the one- 
30 dimensional codes of BDM1 are most strongly correlat- 
ed and are, therefore, oriented at either 0° or 1 80° (step 
2222). If C2 is greater than C1, then BDM2 is selected 
for further processing, because the higher correlation in- 
dicates that BDM2 is oriented at either 0° or 180° (step 
35 2224). Thus, the correct BDM has been found. Before 
determining the address location of the captured image, 
however, computer 1712 must first determine whether 
the selected BDM is at 0° (i.e., oriented correctly), or 
rotated by 180°. 
40 [0052] Fig. 9 is a flowchart showing the steps of how 
computer 171 2 determines the address of the captured 
area of the glyph carpet. Preferably, bit positions along 
a diagonal in the BDM, when the BDM is oriented at 0°, 
have the same value at every other row. The image cap- 
45 ture process and interference from the visual indicia, 
however, may result in errors and erasures in the BDM 
data. To reduce the impact of these errors and erasures, 
computer 1712 performs a majority vote along each di- 
agonal of the odd rows in the direction the odd rows are 
50 staggered, and repeats the majority vote process for 
even rows along the each diagonal in the direction the 
even rows are staggered (step 2225). This results in a 
first code sequence for the odd rows and a second code 
sequence for the even rows. To the extent the majority 
55 vote correctly determines each bit position, the first and 
second code sequences should match a subsequence 
of the original pseudo noise address sequence respec- 
tively corresponding to the odd or even set of rows. 
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[0053] Computer 1712 then retrieves the original 
pseudo noise address code (Get Code 1 ) for rows stag- 
gered forward (step 2226), and retrieves the original 
pseudo noise address code for (Get Code 2) for rows 
staggered backward (step 2228). Using the original 
pseudo noise address code for each code set A and B, 
and the code from the majority voting, computer 1712 
performs four cross correlations (step 2230) to estab- 
lishes the best match of the glyph sequence with the PN 
sequence location for the odd and even rows. 
[0054] More particularly, two adjacent rows of the 
BDM closest to the reference element chosen in step 
221 1 are correlated with the respective complete PN se- 
quences that created the original address carpet. The 
PN sequences could be identical. A forward and back- 
ward correlation is performed for each row. The four cor- 
relations develop four pairs of peak correlation and po- 
sition values: 

1) P1, V1, respectively representing the peak cor- 
relation value and corresponding position for Code 

1 correlated backwards with the complete PN se- 
quence; 

2) Q1 i U1 , respectively representing the peak cor- 
relation value and corresponding position for Code 

2 correlated forwards with the complete PN se- 
quence; 

3) P2, V2, respectively representing the peak cor- 
relation value and corresponding position for Code 

1 correlated forwards with the complete PN se- 
quence; and 

4) Q2, U2, respectively representing the peak cor- 
relation value and corresponding position for Code 

2 correlated backwards with the complete PN se- 
quence. 

[0055] The Uj and V i( position values, where i = 1 or 
2, corresponding to the peak magnitudes are used to 
determine the X and Y values corresponding to the ref- 
erence element chosen in step 2211. Thus, if (P1 + Q1 
> (P2 + Q2) (step 2232), then U1 and V1 are used to 
calculate the X, Y position of the reference glyph loca- 
tion chosen in step 2211 (step 2236). If (P1 + Q1) <= 
(P2 + Q2) (step 2232), then U2 and V2 are used to cal- 
culate the X, Y position of the reference glyph location 
chosen in step 2211 (step 2234). The address informa- 
tion is determined in accordance with the following 
equations: 

X = (V { - Uj + length of full code)/2 



Y = (Vj + Uj - length of full code)/2 

[0056] The calculated X, Y position is then returned 
(step 2238). Note that diagonals correspond to constant 
values of U and V, respectively, while rows and columns 



correspond to constant X and Y Note also that U and V 
could be used directly as address parameters. 
[0057] Thus, an X, Y value associated with the refer- 
ence point chosen in step 2211 has been determined. 
5 Using this information, computer 1712 associates the X, 
Y coordinates with a logical reference, or a combination 
of a logical reference and a control signal (e.g., a button 
click), with a particular operation to be performed. For 
example, the X, Y coordinates could be used as an index 
10 into a table of operations performable by computer 1 71 2 
or other device under direction of computer 1712. The 
X, Y coordinates could be associated with a file open 
command that opens a file associated with an icon lo- 
cated near the X, Y coordinates in the address space. 
is Virtually any operation that can be performed by com- 
puter 1712 could be associated with a particular X, Y 
coordinate, or range of X, Y coordinates. 
[0058] Fig. 13 illustrates an embodiment of a user in- 
terface in which address carpet 12 is divided into hot 
20 zones having respective ranges of addresses corre- 
sponding to a graphic object which in turn corresponds 
to a system object or function. Computer 1 71 2 uses the 
X,Y address determined from the image capture to de- 
termine which area of address carpet 12 was selected 
2S by the user. For example, if the address is X=1 052 and 
Y=32, which is in the upper left area of address carpet 
12, then the user has selected CloseWin.avi. Computer 
1712 utilizes a table that associates X, Y coordinates to 
one or more functions that are to be performed based 
30 on the X, Y coordinates. For example, in the above ex- 
ample, X=1 052 and Y=32 would be associated with the 
one or more functions to be performed when a user se- 
lects CloseWin.avi. Activation could be signaled to com- 
puter 1712 by button 1714, such as a click or multiple 
35 click. Alternatively, activation could be signaled to com- 
puter 1712 by holding the image capture steady for a 
certain time. The precision of location selection is one 
glyph unit and the resolution of differentiated selection 
can be as fine as one glyph unit. Resolution can be ex- 
40 tended to a fraction of a glyph unit provided the capture 
selection device can be referenced to the address car- 
pet to within a fraction of a glyph interval. For example, 
a camera pixel is typically fraction of a glyph interval. 
[0059] Fig. 14 illustrates a user interface 3210 having 
46 several icons that a user can select individually using 
camera pen 1710. User interface 3210 includes numeric 
icon 3238, volume control bar 3250, Undo icon 3246, 
Confirm icon 3240, Duplicate icon 3236, Reset icon 
3244, Dave's Doc icon 321 4, Tom's Doc icon 3216, Jay's 
so Doc icon 3218, File T icon 3232, Folder A icon 3230, 
Outbasket icon 3228, 3.5 Floppy icon 3248, PARC Print- 
er icon 3222, and Tokyo Printer icon 3234. The shaded 
background of user interface 3210 represents an em- 
bedded data code, such as glyphs. In Fig. 14, the em- 
55 bedded data code is an address code. Alternatively, the 
embedded data code could be implemented as a label. 
[0060] To make a selection, the user uses an image 
capture device to select one of the graphic entities and 



8 



15 EP1I 

capture embedded code. This might involve pointing to 
the graphic entity or placing the image capture device 
over the graphic entity, depending on the configuration 
of the graphic reading device (described above). The 
user then activates the selection by pressing a button. 
In response to the button press, the image capture de- 
vice sends a captured image including embedded code 
to a computer, such as computer 1712 of Fig. 17, for 
further processing. 

[0061] User interface 3210 can be implemented on 
any medium capable of having the embedded data- 
bearing image captured therefrom by an image capture 
device, such as a camera pen. For example, user inter- 
face 3210 could be implemented on a computer display 
screen, a piece of paper, or be projected on a wall. Cap- 
ture devices could include, for example, a handheld 
camera, a remote camera, a digital copier platen, or oth- 
er devices capable of capturing an image. 
[0062] The broken lines around each graphic entity 
represent the hot zone for the graphic entity. The hot 
zone is the area within which the user must select in 
order for computer 1712 to get a valid reading that a 
particular graphic entity has been selected. It usually 
comprises the vicinity of the icon. The hot zone is typi- 
cally not shown explicitly to the user, but is shown in Fig. 
14 for purposes of illustration. 

[0063] Fig. 15 illustrates an icon overlaying embed- 
ded codes. The embedded codes in Fig. 1 5 are glyphs. 
This arrangement may be used to implement user inter- 
face 3210 (similar to the embodiment shown in Fig. 16). 
The image capture must be large enough to include suf- 
ficient embedded code to be decodable by computer 
1712. 

[0064] Fig. 1 6 is a simplified block diagram illustrating 
the processes of computer 171 2 that implement userse- 35 
lection and processing of the user selection. More par- 
ticularly, Fig. 16 illustrates the processes of image cap- 
ture, image decoding, code resolving, and syntax de- 
coding of user selections from interface 3210. 
[0065] To activate a graphic entity from user interface 40 
3416, the user uses camera pen 3418 to select the 
graphic entity and presses camera pen button. 3422. In 
one embodiment, camera pen 3418 has two buttons, 
3420 and 3422, that can be used in a manner similar to 
multiple mouse buttons in conventional computer mice. 45 
Computer 1712 responds to clicks from buttons 3420 
and 3422 and interprets the clicks in conjunction with 
the one or more graphic entity selections by the user. 
[0066] In response to the user pressing a button on 
camera pen 3418, camera pen 3418 captures an image 50 
from user interface 341 6, and sends the image to image 
processor 3410. This process includes use of frame 
capture 1728 (not shown in Fig. 16). Each selection 
causes an image of embedded data, and possibly the 
selected icon or a portion of the icon, to be sent to image 5s 
processor 3410 for processing. 

[0067] Selections may also be made from other user 
interfaces. For example, the user may make graphic en- 
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tity selections from a user interface 3426 from a display 
monitor 3424. Thus, a user may select graphic entities 
from paper user interface 3416, electronic display user 
interface 3426, or other user interface, and have the se- 
s lections processed in an integrated manner by computer 
1712. 

[0068] Image processor 341 0 processes the image in 
two steps. First, image processor 341 0 converts the em- 
bedded data in the image to binary values. Second, im- 
10 age processor 3410 analyzes the binary values to de- 
velop an image capture code representing the captured 
image. As long as the image capture code represents 
an area within the hot zone of the particular graphic en- 
tity, computer 1712 will be able to resolve the image cap- 
is ture code into a corresponding selection code repre- 
senting the selected computer entity or operation within 
the hot zone. After determining the image capture code, 
image processor 3410 passes the image capture code 
to code resotver 3412. 
20 [0069] Code resolver 3412 resolves the image cap- 
ture code into a selection code corresponding to a com- 
puter system entity or operation represented by the 
graphic entity selected by the user. To determine which 
graphic entity is selected, code resolver 3412 uses a 
25 mapping table (not shown) that correlates an image cap- 
ture code to a selection code representing the selected 
graphic entity. For example, for user interface 3210 of 
Fig. 1 4, the mapping table maps all image capture codes 
within the hot zone for Dave's Doc icon 321 4 to a selec- 
30 tion code representing Dave's Doc icon 3214. Similarly, 
the mapping table maps all image capture codes within 
the hot zone for Undo icon 3246 to the selection code 
representing the Undo operation. Thus, code resolver 
3412 determines which computer system entity or op- 
eration corresponds to the image capture code by using 
the mapping table, similar to the process described 
above with respect to Fig. 16. Code resolver 3412 then 
passes the selection code representing the selected 
graphic entity to syntax processor 3414. 
[0070] In one embodiment, labels are used instead of 
address codes. In a label system, the image portion 
within a hot zone will always decode into a particular 
label (i.e., image capture code) corresponding to the se- 
lected graphic entity. It is possible to implement the la- 
bels so that there is no need for code resolver 3412 to 
perform the mapping step. In such a system, the label 
itself sufficiently identifies the computer system entity or 
operation. 

[0071] Syntax processor 3414 receives selection 
codes from code resolver 3412, and outputs an opera- 
tion code to computer 1712 when a particular combina- 
tion of selection codes having a particular syntax is re- 
ceived. If a particular selection code or combination of 
selection codes is received, representing a particular 
graphic entity selection or combination of graphic entity 
selections by the user, syntax processor directs compu- 
ter 1712 to perform an operation corresponding to the 
selection code or combination of selection codes. 
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[0072] Image processor 3410, code resolver 3412 
and syntax processor may be implemented as hardware 
software, or a combination of hardware and software. If 
implemented in whole or in part as software elements, 
those software elements may be stored on a disk, or s 
transferred over a network as a signal. Software ele- 
ments implementing all or part of the apparatus and 
methods consistent with the invention as disclosed 
herein may be stored as program components or mod- 
ules on a computer readable and usable medium. 10 
[0073] Computer 1712 may provide feedback to the 
user at appropriate times during the processing of se- 
lections by the user (not shown). The feedback may be 
any type of sensory feedback, such as audio, visual, or 
tactile, similar to conventional systems. For example, if is 
a computer screen is connected to computer 1712, 
feedback may be visual, such as by highlighting or ani- 
mation. Feedback may be used, for example, as part of 
syntax processing to notify the user that a particular se- 
lection has been made, or a particular series of selec- 20 
tions have been made. 

[0074] Multi-level selection involves user selection of 
a series of graphic entities that, when interpreted togeth- 
er by computer 1712, causes computer 1712 to perform 
an operation or select a computer system entity. Using 25 
multi-level selection, when the user wants computer 
171 2 to perform an operation, the user selects a series 
of graphical entities on user interface 3210 by sequen- 
tially placing camera pen 3418 to select individual 
graphic entities of the interface, and pressing button 30 
3422 to select each entity. Each time button 3422 is 
pressed, the image under the tip of camera pen 341 8 is 
sent to computer 1712. Image processor 341 0 receives 
each image and decodes the embedded data in the im- 
age to form an image capture code corresponding to the 35 
capture, code resolver 3412 maps the image capture 
code into a selection code, and syntax processor 3414 
analyzes the received selection codes as they are re- 
ceived, and performs an operation when the particular 
combination of codes received has appropriate syntax 40 
and values. 

[0075] For example, if a user first selects David's Doc 
icon 3214 and then PARC Printer icon 3222, computer 
1712 prints the document associated with David's Doc 
icon 3214 on the printer associated with PARC Printer 45 
icon 3222. Other operations are performed in response 
to other selections of graphic entities. For example, a 
document icon may be selected, followed by selection 
of Outbasket icon 3228, to place the document in the 
electronic outbasket file in computer 1712 that is asso- so 
ciated with the Outbasket icon 3228. Document icon se- 
lection might be followed by selection of Folder A icon 
3230, to copy the document to a an electronic folder in 
computer 1712 that is associated with Folder A icon 
3230. Alternatively, a document may be selected, fol- 55 
lowed by selection of File T icon 3232, to copy the doc- 
ument to an electronic file in computer 1712 that is as- 
sociated with File T icon 3232. 



[0076] Selection of certain graphic entities in user in- 
terface 3210 augments or assists in the functionality of 
user interface 3210. For example, computer 1712 can 
be programmed to respond to selection of graphic enti- 
ties that are typical menu items on conventional graph- 
ical user interfaces. Thus, selecting Undo icon 3246 al- 
lows the user to undo one or more levels of selection or 
operations. Selection of Confirm icon 3240 allows the 
user to confirm information if requested to do so. For 
example, computer 1712 might request confirmation of 
a requested operation by outputting a voice on a speak- 
er requesting confirmation. Selection of Reset icon 3244 
allows the user to delete all previous selections. Selec- 
tion of a document icon, followed by selection of Dupli- 
cate icon 3236 duplicates the selected document. 
[0077] Numeric icon 3238 allows a user to enter nu- 
merical information associated with the operations to be 
performed by computer 1712. For example, to enter 
"22, 0 the user selects D 2," "2," and ■#■ (selecting ■#■ in- 
dicates end of selection). Thus, if the user wants 22 cop- 
ies of Jay's Doc printed on the Tokyo printer, the user 
selects ■2," "2,° °#," Jays Doc icon 3218, and Tokyo 
Printer icon 3234. Computer 1712 receives the image 
for each selection and decodes the embedded data cap- 
tured in the image into selection codes representing the 
selections "2 f B "2," "#," Jay's Doc icon 3218, and Tokyo 
Printer icon 3234. Computer 1712 interprets this series 
of selection codes to mean that the user wants 22 copies 
of the document Jay's Doc printed on the Tokyo printer, 
and prints the copies on the Tokyo printer. In another 
embodiment, an entire keyboard or multiple alternative 
keyboards could be provided as printed graphical user 
interfaces. 

[0078] By selecting a position along Volume control 
bar 3250, a user may select the level of speaker volume 
for computer 1 71 2. A graphic entity such as Volume con- 
trol bar 3250 takes advantage of the fine level of position 
selection available with an embedded code. For exam- 
ple, if a glyph address carpet is used, each glyph of the 
address carpet has its own unique address. Thus, Vol- 
ume bar 3250 can take advantage of this fine level of 
detail to provide a practically infinite range of volume 
level values from Hi to Lo along the vertical direction. 
[0079] Computer 1712 performs operations in re- 
sponse to a single user selection or series of user se- 
lections. Operations are also performed in response to 
multiple selections of the same graphic entity, either 
alone or in combination with other selections. 
[0080] In summary, computer 1712 can be pro- 
grammed to respond to individual or combinations of 
graphic entity selections to perform corresponding func- 
tions performable by the computer. Additionally, compu- 
ter 1712 can be programmed to perform operations in 
response to one or more button signals from camera 
pen 341 8, or signals from other user input devices. Each 
input device, including camera pen 3418, may have one 
or more buttons, and each button may be interpreted 
distinctly as a separate signal. 
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[0081] Fig. 17 is a flowchart showing the processing 
performed by code resolver 3412 when receiving a se- 
lection from the user. Code resolver 3412 first waits for 
a signal indicating that button 3422 on camera pen 341 8 
has been pressed (step 3510). In response to the signal, s 
code resolver 3412 captures and decodes the image 
from camera pen 3418 into an image capture code (step 
351 2). If the image is not decodable (step 351 4), code 
resolver 3412 may signal the user that the selection is 
not decodable (step 3516), and processing continues 10 
by waiting for the next signal indicating that button 3422 
on camera pen 3418 has been pressed (step 3510). 
[0082] If the image is decodable (step 351 4), a selec- 
tion confirmation is preferably provided to the user (step 
351 8), the image capture code is mapped to a selection is 
code, and the selection code is sent to syntax processor 
3414, for storage with other unresolved pending selec- 
tion codes, if any (step 3520). Syntax processor 341 4 
buffers the selection codes until the pending selection 
codes represent an operation to be performed. 20 
[0083] Fig. 1 8 is a flow chart showing the processing 
performed by syntax processor 341 4. Syntax processor 
3414 first waits until a new selection code is received 
from code resolver 3412 (step 3610). When a new se- 
lection code is received, syntax processor 3414 stores 2S 
the new selection code with pending selection codes 
from previous selections, if any, in the order received. 
Syntax processor 3414 then analyzes and parses the 
pending selection codes to determine whether the pend- 
ing selection codes collectively or individually indicate 30 
that any operations should be performed (step 3612). 
Syntax processor 3414 analyzes the pending selection 
codes for sequences or combinations of selection codes 
that meet predetermined criteria indicating that an op- 
eration should be performed. 35 
[0084] Syntax processor 3414 also analyzes the 
pending selection codes to determine whether the 
codes should be restructured (step 361 4). Restructuring 
may involve changing the order of pending selection 
codes or deleting pending selection codes. One or more 40 
pending selection codes may be removed if the selec- 
tion codes indicate a selection sequence or combination 
that requires removal of codes based on the syntax. For 
example, a system may be implemented where a user 
selects a first document icon, using camera pen 3418 46 
and then performs an inclusive selection of a second 
document icon by selecting the second document icon 
and double clicking camera button 3422. In this case, 
the sequence of codes would be recognized as an in- 
clusive selection, and syntax processor 3414 would so 
keep both codes pending. If, however, the first docu- 
ment icon is selected, and followed by a non-inclusive 
selection of a second document icon, restructuring step 
361 4 would drop the code corresponding to the first doc- 
ument icon because the selection was non -inclusive, ss 
This is equivalent to clicking on a first icon, and then 
clicking on a second icon, for non-inclusive selection in 
some conventional systems. 



[0085] If syntax processor 3414 determines at step 
3612 that the pending selection codes do not indicate 
an operation, syntax processor 3414 returns to waiting 
for new selection codes (step 3616). If syntax processor 
341 4 determines at step 361 2 that the pending selection 
codes represent an operation (step 361 6), the operation 
is performed (step 3618), and syntax processor 3414 
again begins waiting for the next selection code (step 
3610). 

[0086] Fig. 19 is a block diagram of an example of a 
state diagram that may be implemented by syntax proc- 
essor 3414 when performing the flow chart of Fig. 18. 
Syntax processor 3414 starts in a WAIT state 3710. As 
long as no selection codes are received, syntax proces- 
sor 341 4 remains in WAIT state 371 0. If syntax proces- 
sor 3414 receives a document selection code, syntax 
processor 3414 moves to the DOC state 3712. If a se- 
lection code other than a number or printer selection 
code is received, syntax processor 3414 returns to 
WAIT state 371 0. If a number selection code is received, 
however, syntax processor 3414 moves to DOC, NBR 
state 3716, indicating that a document selection code 
and number selection code have been received. If a 
printer selection code is received, syntax processor 
341 4 moves to the PRINT state 371 8 to print the select- 
ed document. After printing is finished, syntax processor 
3414 returns to WAIT state 3710. 
[0087] If syntax processor 3414 is in WAIT state 3710 
and a number selection code is received, syntax proc- 
essor 341 4 moves to NBR state 371 4. If a selection code 
other than a document selection code is received, syn- 
tax processor 341 4 returns to WAIT state 371 0. If a doc- 
ument selection code is received, however, syntax proc- 
essor 3414 moves to DOC, NBR state 3716. 
[0088] From DOC, NBR state 371 6, if a selection code 
other than a printer selection code is received, syntax 
processor 3414 moves to WAIT state 3710. If a printer 
selection code is received, however, syntax processor 
3414 moves to PRINT state 3718 to print the document 
the specified number of times on the specified printer. 
After printing is finished, syntax processor 3414 returns 
to WAIT state 3710. 

[0089] The state diagram of Fig. 1 9 is only represent- 
ative of a state diagram that could be used by syntax 
processor 341 4 to process incoming selection code sig- 
nals. The complexity and particular syntax processing 
will change from system to system. Embodiments might 
also include moving between states based on device 
buttons or other user selections in addition to selection 
codes. 

[0090] Figs. 20 and 21 illustrate user interfaces 3810 
and 3812. User interfaces 3810 and 3812 are similar to 
user interface 3210, but include context icons. The con- 
text icons direct computer 1 71 2 to operate in respective- 
ly different contexts when processing user interface 
3810 or 3812. Context for user interface 3810 is select- 
ed using Dave's Desktop #3 icon 3820, whereas context 
for user interface 3812 is selected using a Dave's Desk- 
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top #4 .con. When selected, each context icon causes 
computer 1712 to use respective combinations of map- 
ping and syntax processing modules. The context icons 
are graphically associated with an apparent area con- 
taining embedded data code that must be substantially s 
captured to provide the context code. Partial capture 
may be enabled by error correction codes. For example 
the visible boundary marking on the context icon of Figs' 
20 and 21 is an example of the apparent area 
[0091] To select Dave's Desktop #3 icon 3820, the us- io 
er uses camera pen 3418 to select Dave's Desktop #3 
icon 3820 as described above with respect to the graph- 
ic entities of Fig. 14. After Dave's Desktop #3 icon 3820 
is selected, the remaining selections from user interface 
381 0 will be mapped and processed syntactically based is 
on the context code derived from Dave's Desktop #3 
icon 3820. Similarly, after Dave's Desktop #4 icon 3820 
is selected, the remaining selections from user interface 
381 2 will be mapped and processed syntactically based 
on the context code derived from Dave's Desktop #4 20 
icon 3820. 

[0092] Figs. 22, 23, 24, and 25 show respective logi- 
cal representations of embodiments of code resolver 
341 2 and syntax processor 3414. Fig. 22 illustrates an 
embodiment wherein code resolver 341 2 includes mul- 2s 
tiple mapping modules 3910 and multiple syntax 
processing modules 3912 corresponding to respective 
contexts. Thus, when a user selects a context graphic 
entity, the corresponding context mapping module of 
code resolver 3412 and syntax processing module of so 
syntax processor 341 4 are used from that point on until 
the context is changed by the system or the user For 
example, if context icon A is selected, from that point on 
context A mapping module and context A syntax 
processing module are used. 35 
[0093] Fig. 23 illustrates a variation of Fig. 22. In Fig 
23, mapping modules 391 0 and syntax processing mod- 
ules 3916 are independently selected. For example a 
user might select context A mapping module, and con- 
text E syntax processing module. This embodiment is 40 
in contrast to the embodiment of Fig. 22, where sets of 
mapping and syntax processing modules are selected 
by a single context code. 

[0094] Fig. 24 illustrates another embodiment of code 
resolver 3412 and syntax processor 3414. In this em- 45 
bodiment, mapping is always performed by mapping 
module 3918, irrespective of context. Syntax processor 
3414, however, is context selectable to provide syntax 
processing by one of syntax processing modules 3912 
as selected by the user or the system. so 
[0095] Fig. 25 illustrates another embodiment of code 
resolver 341 2 and syntax processor 341 4 wherein map- 
ping context can be chosen by the user or system, but 
syntax processing is performed the same, irrespective 
of context, by syntax processing 3920. ' 55 

[0096] The embodiment shown in Fig. 14 represents 
a fixed context system. This embodiment comprises a 
single mapping module in code resolver 3412 and a sin- 



gle syntax processing module in syntax processor 3414 
(not shown). 

[0097] Changing context allows a much broader 
range of functionality and flexibility, allowing operations 
performed in response to a particular selection of graph- 
ic entities to vary from context to context. For example 
multiple user interfaces may each utilize the same em- 
bedded code address space, but by providing each with 
a unique context, the mapping for each respective user 
interface can be entirely different. 
[0098] Moreover, context may be selected by virtually 
any form of user input or by the system. For example, 
an embedded code user interface could be used in con- 
junction with a web page. The web page would provide 
the context for mapping, syntax, or both, for the user 
interface. 

[0099] Fig. 26 illustrates an example of a state dia- 
gram that may be used to implement a context-based 
user interface, such as shown in Figs. 20 and 21 . The 
upper half of the state diagram represents a first context 
A, and the lower half of the state diagram represents a 
second context B. State diagram elements WAIT state 
3710, DOC state 3712, IMBR state 3714, DOC NBR 
state 3716 and PRINT state 3718 are the same states 
as shown in Fig. 19. 

[01 00] Beginning at WAIT state 4020, syntax proces- 
sor 3414 waits for a context selection code from a user. 
If the user selects a context icon, such as Dave's desk- 
top #3 icon 3820, indicating context A, syntax processor 
3414 moves from WAIT state 4020 to WAIT state 3710. 
Processing continues as described above in connection 
with Fig. 19. Thus, as in Fig. 19, the syntax required for 
context A is that a combination of document/number/ 
printer, number/document/printer, or document/printer 
be selected. 

[0101] While in WAIT state 4020 or WAIT state 3710, 
if the user selects a context icon indicating context B, 
syntax processor 3414 moves to WAIT state 4022. In 
context B, user input is processed according to the syn- 
tax required by WAIT state 4022, DOC state 4028 NBR 
state 4024 and PRINT state 4026. From WAIT state 
4022, if the user selects a number, syntax processor 
3414 moves to NBR state 4024. If a document selection 
code is received, syntax processor 3414 moves to state 
DOC 4028. If the user then selects a number, syntax 
processor 3414 moves to PRINT state 4026, causing 
computer 1712 to print the document, and returns to 
WAIT state 4022. 

[01 02] If syntax processor 341 4 is at WAIT state 4022, 
and a number is received, syntax processor 3414 
moves to NBR state 4024. If a document selection code 
is then received, syntax processor 3414 moves to 
PRINT state 4026, the document is printed and syntax 
processor 3414 returns to WAIT state 4022. Note also 
that from WAIT state 4022, if a context A selection code 
is received, processor 1712 moves to WAIT state 3710 
[01 03] In summary, context A requires a combination 
of document/number/printer, number/document/printer 
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or document/printer be selected. Context B merely re- 
quires document/number or number/document. Thus, in 
context B, the printer is already chosen, so computer 
1712 will print the document after only receiving a doc- 
ument and number. In context A, however, the printer s 
must be selected. 

[0104] The state diagrams of Fig. 1 9 and Fig. 26 rep- 
resent simplified syntax processing. Systems will typi- 
cally have more complex state diagrams that can be im- 
plemented for any combination of user selections to re- 10 
suit in selection of a computer function. 
[0105] Fig. 27 illustrates a user interface embodiment 
similar to the interface shown in Figs. 32, 20, and 21 , 
but is implemented using a visual border block around 
the graphic entity instead of implicit hot zones, which is 
are not visible. When a user places the tip of camera 
pen 3418 within the border, an image of embedded data 
will be captured when a button is pushed that will yield 
a selection code corresponding to the graphic entity 
within the block. 20 
[0106] The context-based system of Figs. 20, 21 and 
41 may also be implemented using an embedded con- 
text code, thus obviating the need for an explicit local- 
ized context icon. In one embodiment, context is imple- 
mented by a coding system that embeds context codes 2s 
with location codes in the embedded data. Thus, when 
computer 1712 decodes a captured image, the decod- 
ing provides both a location and a context. Such a cod- 
ing system may be called a distributed labeling code. 
For example, see Figs. 10, 11, and 12. User interface 30 
embodiments similar to those shown in Fig. 27 may also 
be implemented without context icons by embedding 
one or more context codes in the coding system. Con- 
text codes and address codes may be interleaved using 
an approach similar to that shown in Fig. 5. 35 
[0107] Fig. 28 is a block diagram showing the use of 
multiple sections of user interface, each on a separate 
substrate. The embodiment also illustrates grouping 
icons by related graphic entities. User interface 4210 is 
comprised of graph ic entities that augment the selection 40 
process, user interface 4212 carries document graphic 
entities, user interface 4214 includes document storage 
graphic entities, user interface 4216 comprises input/ 
output device graphic entities, and user interface 4218 
includes context A icon 4220 and context B icon 4222. 4S 
A user may make selections from any of the user inter- 
face sections shown in Fig. 28 to cause computer 1712 
to perform operations. Thus, Fig. 28 illustrates the flex- 
ible nature of the image capture user interface disclosed 
herein. Physically, the user interface sections could be so 
different types of substrates, as in paper prints, electron- 
ic screen, and image projection on a wall or screen. 
[0108] Moreover, each user interface portion could be 
implemented using different types of embedded codes. 
For example, one portion could be implemented using ss 
label codes, whereas another portion might be imple- 
mented using address carpets. Additionally, a single 
portion of user interface could incorporate multiple types 



of embedded data codes. 

[0109] Interface portions may include superpositions 
of embedded data printing and graphical element print- 
ing, such as multi-color printing, image projection on a 
print, or superposition of multiple projected images. 
Care must be taken to establish and maintain a logical 
and spatial registration of the superimposed images. 
[0110] Fig. 29 is a block diagram illustrating multilevel 
selection in a document processing environment. Doc- 
ument 4314 is comprised of text overlaying embedded 
data, as indicated by the background gray area. The us- 
er selects text by first making a selection at the begin- 
ning of the selection of interest (indicated by - 1 B ), and 
then making a second selection at the end of the portion 
of interest, in this case the end of the sentence (indicat- 
ed by °2 H ). To print, the user then selects either PARC 
Printer icon 3222 or Tokyo Printer icon 3234, which re- 
sults in printing out the selected text on the selected 
printer. Other variations of this general multilevel selec- 
tion scheme illustrated in Fig. 29 may be utilized for oth- 
er text and document processing operations. 
[0111] Document processing can also take advantage 
of the fine level of selection offered by address carpets 
and other embedded data. Because eachfccation is de- 
coded into a different code, a user can select a fine level 
of detail, typically one glyph in the case of glyph address 
carpets. On a 300 dpi print, glyph spacing may be, for 
example, 60/inch, which is comparable to conventional 
screen resolution. Thus, text selections can be per- 
formed using camera pen 3418 which mimic the opera- 
tions performed by a conventional mouse and electronic 
user interface. Moreover, interpolation between glyphs 
can extend selection resolution. 

[0112] Fig. 30 illustrates a user interface example 
showing an e-mail system using local embedded data 
code labels. In the embodiment shown, an organization- 
al structure of a company is provided with nodes se- 
lectable by local embedded data code labels. For exam- 
ple, node 4410 represents supervisor 5 that may be se- 
lected by embedded data label 441 2. The user interface 
provides a mechanism for selecting recipients for e-mail 
documents. A user could select Dave's Doc icon 3214 
from Fig. 14, and then node 4410 using embedded data 
label 4412 to direct the document Dave's Doc to the su- 
pervisor associated with node 4410. Icon 4414 is used 
for selecting the context associated with the user inter- 
face. Alternatively to the local embedded data code la- 
bels, or in addition to them, the user interface of Fig. 30 
may be implemented using address carpets as a back- 
ground. 



Claims 



1. 



Apparatus for use with a user interface having sen- 
sory indicia and embedded data code, and an im- 
age capture device for selecting a portion of the us- 
er interface and sending image information corre- 
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sponding to the portion, comprising: 

an image processor for decoding first image in- 
formation into a first image capture code; 
a resolver for mapping the first image capture s 
code into a first selection code; and 
a syntax processor for analyzing the first selec- 
tion code with a second selection code. 

2. A method for use with a user interface having sen- 10 
sory indicia and embedded data code, and an im- 
age capture device for selecting a portion of the us- 
er interface and sending image information corre- 
sponding to the portion, comprising: 

is 

decoding first image information into a first im- 
age capture code; 

mapping the first image capture code into a first 
selection code; and 

analyzing the first selection code with a second 20 
selection code. 

3. A graphical user interface for use with a user inter- 
face system including a device for capturing a por- 

tion of the graphical user interface in response to 25 
selection of sensory indicia, and a computer system 
responsive to embedded code in the portion for per- 
forming an operation related to the sensory indicia 
and a first selection code, comprising: 



sensory indicia; and 

embedded data codes having a predetermined 
spatial relationship to the sensory indicia. 
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IN 1970, XEROX CORPORATION GATHERED TOGETHER 
A TEAM OF WORLD CLASS RESEARCHERS AND GAVE 
THEM THE MISSION OF CREATING "THE 
ARCHITECTURE OF INFORMATION.0THE SCIENTISTS 
OF THE PALO ALTO RESEARCH CENTER (PARC) 
LIVED UP TO THIS CHALLENGE BY INVENTING 
PERSONAL DISTRIBUTED COMPUTING, GRAPHICAL 
USER INTERFACES, THE FIRST COMMERCIAL MOUSE, 
BIT-MAPPED DISPLAYS, ETHERNET, CLIENT/SERVER 
ARCHITECTURE, OBJECT-ORIENTED PROGRAMMING, 
LASER PRINTING AND MANY OF THE BASIC 
PROTOCOLS OF THE INTERNET. IN THE YEARS SINCE 
XEROX ESTABLISHED THE PALO ALTO RESEARCH 
CENTER-PARC TECHNOLOGIES HAVE CHANGED THE 
WORLD.©PRESENTED HERE ARE SOME OF THE 
HIGHLIGHTS OF PARC'S WORK. 
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